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Program Description^ 

Peer-Assisted Learning Strategies is a peer-tutoring program for grades 
K-6 that aims to improve student proficiency in math and other disci- 
plines. This report focuses on Peer-Assisted Learning Strategies for math. 

The math program supplements students’ existing math curriculum and 
is based on peer-mediated instruction, a process whereby students work 
in pairs or smail groups to tutor each other. During tutoring sessions, 
students work together on worksheets that target specific math skills, with 
one student designated to correct his or her partner’s errors, award points 
for correct responses, and provide consistent encouragement and feed- 
back. The program uses videos and teacher-provided scripted instruction 
to train students to engage in peer tutoring. Developers recommend that 
students participate in peer-tutoring sessions two to three times a week 
for approximateiy 30 minutes per session. 

Research^ 

The What Works Ciearinghouse (WWC) identified one study of Peer- 
Assisted Learning Strategies that both fails within the scope of the Elementary School Mathematics topic area and 
meets WWC evidence standards. This study meets WWC evidence standards without reservations and included 
328 elementary school students in the first grade in five schools in the southeastern United States. 
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The WWC considers the extent of evidence for Peer-Assisted Learning Strategies on the math performance of 
elementary school students to be small for the mathematics achievement domain, the only outcome domain exam- 
ined for studies reviewed under the Elementary School Mathematics topic area. 


Effectiveness 

Peer-Assisted Learning Strategies was found to have no discernible effects on mathematics achievement for 
elementary school students. 


Table 1. Summary of findings^ 




Improvement index 

(percentiie points) 




Outcome domain 

Rating of effectiveness 

Average 

Range 

Number of 
studies 

Number of 
students 

Extent of 
evidence 

Mathematics achievement 

No discernible effects 

+2 

-1 to +6 

1 

328 

Small 


Peer-Assisted Learning Strategies January 2013 


Page 1 









WWC Intervention Report 


Program Information 

Background 

Developed by Doug and Lynn Fuchs, Peer-Assisted Learning Strategies is distributed by the Vanderbilt Kennedy 
Center and the Department of Special Education, Peabody College, Vanderbilt University. Address: PALS Outreach, 
Vanderbilt University, Peabody Box 228, 110 Magnolia Circle, Nashville, TN 37203-5701. Email: PALS@vanderbilt. 
edu. Web: http://kc.vanderbilt.edu/pals. Telephone: (615) 343-4782. 

Program details 

Peer-Assisted Learning Strategies uses two peer-tutoring procedures, referred to as coaching and practice. During 
coaching, students work on problems in the skill area (e.g., adding, subtracting with regrouping, number concepts, 
charts, and graphs) to which they have been assigned. One of the students serves as the coach (peer tutor) and one 
serves as the tutee. The tutoring sessions are organized around completion of student worksheets that contain a 
series of questions, differing by problem type (e.g., addition or subtraction). Coaches correct errors when the tutee 
gets an answer wrong. Coaching usually lasts 15-20 minutes. Materials are available for students in grades K-6. 

During the practice portion of peer-tutoring, students work independently on a mixed-problem worksheet contain- 
ing problems of the type just completed, then exchange papers and score each other’s practice sheets. Practice 
lasts 5-10 minutes. 

Teachers select student groups by identifying student’s strengths and weaknesses and put students together who 
will best learn from each other. Groups are changed regularly, and as students work on a variety of skills over time, 
all students have the opportunity to serve as coaches (i.e., the students charged with asking questions and correct- 
ing errors). 


Cost 

Peer-Assisted Learning Strategies materials range from $15 to $40. The Kindergarten Math Manual ($40) includes 
training scripts and other materials for 16 Peer-Assisted Learning Strategies lessons. The First Grade Math Manual 
($40) includes training scripts and student game boards for 18 Peer-Assisted Learning Strategies lessons. The 
Grades 2-6 Math Manual ($40) includes training scripts and sample student materials. Student materials for grades 
2-6 must be purchased separately ($30 for each grade). The cost of a one-day training workshop is estimated at 
$1 ,500 plus travel expenses for the presenter. 
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Research Summary 


The WWC identified 13 studies that investigated the effects of Peer- 
Assisted Learning Strategies on the math performance of eiemen- 
tary schooi students. 

The WWC reviewed all of those studies against group design 
evidence standards. One study (Fuchs, Fuchs, Yazdian, & Powell, 
2002) is a randomized controlled trial that meets WWC evidence 
standards without reservations. The study is summarized in this 
report. Four studies do not meet WWC evidence standards. The 
remaining eight studies do not meet WWC eligibility screens for 
review in this topic area. Citations for all 13 studies are in the Refer- 
ences section, which begins on p. 5. 


Table 2. Scope of reviewed research^ 


Grade 

1 

Delivery method 

Small group 

Program type 

Supplement 

Studies reviewed 

13 studies 

Group design studies that meet 
WWC evidence standards 

• without reservations 

• with reservations 

1 study 
0 studies 


Summary of study meeting WWC evidence standards without reservations 

Fuchs et al. (2002) conducted a randomized controlled trial in which 20 first-grade classrooms in five schools 
from a metropolitan public school district in the southeastern United States were randomly assigned to either the 
Peer-Assisted Learning Strategies intervention condition or a comparison condition. The 10 teachers assigned to 
the intervention condition implemented Peer-Assisted Learning Strategies as a supplement to the district’s core 
curriculum. Math Advantage. The program was implemented in 30-minute sessions, three times a week, for 16 
weeks. The study authors wanted to ensure that students in the intervention and comparison groups received the 
same amount of mathematics instruction during the 1 6 weeks when Peer-Assisted Learning Strategies was used 
with the intervention group. Therefore, the study’s implementation of Peer-Assisted Learning Strategies replaced 
some mathematics instruction activities that intervention teachers would otherwise have used. The 10 teachers 
in the comparison condition used Math Advantage Grade 1 in their classrooms, which includes teacher-directed 
instruction and student work with manipulatives and worksheets. The final analysis sample included 328 first-grade 
students (162 Peer-Assisted Learning Strategies students and 166 comparison students). 


Summary of studies meeting WWC evidence standards with reservations 

No studies of Peer-Assisted Learning Strategies meet WWC evidence standards with reservations. 
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Effectiveness Summary 

The WWC review of Peer-Assisted Learning Strategies for the Elementary School Mathematics topic includes 
student outcomes in one domain: mathematics achievement. The findings beiow present the authors’ estimates 
and WWC-caicuiated estimates of the size and statistical significance of the effects of Peer-Assisted Learning 
Strategies on the math performance of elementary school students. For a more detailed description of the rating of 
effectiveness and extent of evidence criteria, see the WWC Rating Criteria on p. 1 3. 

Summary of effectiveness for the mathematics achievement domain 

One study reported findings in the mathematics achievement domain. 

Fuchs et al. (2002) reported statisticaily significant effects of Peer-Assisted Learning Strategies on mathemat- 
ics achievement based on the items of the Stanford Achievement Test (SAT) aiigned with Peer-Assisted Learning 
Strategies and no significant effects of the program on mathematics achievement based on the items of the SAT 
unaligned with Peer-Assisted Learning Strategies. After accounting for ciustering and multiple comparisons, the 
WWC determined that none of the findings were statisticaliy significant or large enough to be considered substan- 
tively important according to WWC criteria. The WWC characterizes these study findings as an indeterminate effect. 

Thus, for the mathematics achievement domain, one study had an indeterminate effect. This results In a rating of no 
discernible effects, with a small extent of evidence. 


Table 3. Rating of effectiveness and extent of evidence for the mathematics achievement domain 


Rating of effectiveness 

Criteria met 

No discernible effects 

No affirmative evidence 
of effects. 

In the one study that reported findings, the estimated impact of the intervention on outcomes in the mathematics 
achievement domain was neither statistically significant nor large enough to be substantively important. 

Extent of evidence 

Criteria met 

Small 

One study that included 328 students in five schools reported evidence of effectiveness in the mathematics 
achievement domain. 


Peer-Assisted Learning Strategies January 2013 


Page 4 




WWC Intervention Report 


References 

Study that meets WWC evidence standards without reservations 

Fuchs, L S., Fuchs, D., Yazdian, L, & Powell, S. R. (2002). Enhancing first-grade children’s mathematical develop- 
ment with Peer-Assisted Learning Strategies. School Psychology Review, 37(4), 569-583. 

Studies that meets WWC evidence standards with reservations 

None 

Studies that do not meet WWC evidence standards 

Codding, R. S., Chan-lannetta, L., George, S., Ferreira, K., &Volpe, R. (2011). Early number skills: Examining the 
effects of class-wide interventions on kindergarten performance. School Psychology Quarterly, 26(1), 85-96. 
The study does not meet WWC evidence standards because the measures of effectiveness cannot be attrib- 
uted solely to the intervention— there was only one unit assigned to one or both conditions. 

Fuchs, L. S., Fuchs, D., Hamlett, C. L., Phillips, N. B., Karns, K., & Dutka, S. (1997). Enhancing students’ helping 
behavior during peer-mediated instruction with conceptual mathematical explanations. The Elementary School 
Journal, 97(3), 223-249. The study does not meet WWC evidence standards because it uses a quasi-experi- 
mental design in which the analytic intervention and comparison groups are not shown to be equivalent. 

Fuchs, L. S., Fuchs, D., & Karns, K. (2001). Enhancing kindergartners’ mathematical development: Effects of peer- 
assisted learning strategies. The Elementary School Journal, 101(b), 495-510. The study does not meet WWC 
evidence standards because the measures of effectiveness cannot be attributed solely to the intervention — 
the intervention was combined with another intervention. 

Fuchs, L. S., Fuchs, D., Phillips, N. B., Hamlett, C. L., & Karns, K. (1995). Acquisition and transfer effects of class- 
wide peer-assisted learning strategies in mathematics for students with varying learning histories. School 
Psychology Review, 24(4), 604-620. The study does not meet WWC evidence standards because it uses a 
quasi-experimental design in which the analytic intervention and comparison groups are not shown to be 
equivalent. 

Studies that are ineligible for review using the Elementary School Mathematics Evidence Review Protocol 

Calhoon, M. B., & Fuchs, L. S. (2003). The effects of peer-assisted learning strategies and curriculum-based mea- 
surement on the mathematics performance of secondary students with disabilities. Remedial and Special 
Education, 24(4), 235-245. The study is ineligible for review because it does not use a sample aligned with the 
protocol— the sample includes more than 60% learning disabled students. 

Fuchs, L. S., Fuchs, D., Hamlett, C. L., Phillips, N. B., & Karns, K. (1995). General educators’ specialized adaptation 
for students with learning disabilities. Exceptional Children, 61(b), 440-459. The study is ineligible for review 
because it does not use a sample aligned with the protocol— the sample includes more than 60% learning 
disabled students. 

Ginsburg-Block, M. D., Rohrbeck, C. A., & Fantuzzo, J. W. (2006). A meta-analytic review of social, self-concept, 
and behavioral outcomes of peer-assisted learning. Journal of Educational Psychology, 98(4), 732-749. The 
study is ineligible for review because it is a secondary analysis of the effectiveness of an intervention, such as 
a meta-analysis or research literature review. 

Harris, A. A. (2009). Comparing effects of two grouping conditions to teach algebraic problem-solving to stu- 
dents with mild disabilities in inclusive settings. Dissertation Abstracts International, 70(05A), 113-1618. 
(AAI3356479) The study is ineligible for review because it does not use a sample aligned with the protocol — 
the sample includes more than 60% learning disabled students. 

Rutherford, L. E., Dupaul, G. J., & Jitendra, A. K. (2008). Examining the relationship between treatment outcomes 
for academic achievement and social skills in school-age children with attention-deficit hyperactivity disorder. 


Peer-Assisted Learning Strategies January 2013 


Page 5 


WWC Intervention Report 


Psychology in the Schools, 45(2), 145-157. The study is ineligible for review because it does not use a sampie 
aligned with the protocol— the sample includes more than 60% learning disabled students. 

Ryan, J. B., Pierce, C. D., & Mooney, P. (2008). Evidence-based teaching strategies for students with EBD. Beyond 
Behavior, 17(3), 22-29. The study is ineiigible for review because it does not use a sample aligned with the 
protocol— the sample includes more than 60% learning disabied students. 

Stecker, P. M., Fuchs, L. S., & Fuchs, D. (2005). Using curricuium-based measurement to improve student achieve- 
ment: Review of research. Psychology in the Schools, 42(8), 795-819. The study is ineiigible for review 
because it is a secondary analysis of the effectiveness of an intervention, such as a meta-analysis or research 
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Appendix A: Research detaiis for Fuchs et ai. (2002) 

Fuchs, L. S., Fuchs, D., Yazdian, L., & Powell, S. R. (2002). Enhancing first-grade children’s mathematical 
development with Peer-Assisted Learning Strategies. School Psychology Review, 3f(4), 569-583. 

Tahie A. Summary of findings Meets WWC evidence standards without reservations 


study findings 
Average improvement index 

Outcome domain Sample size (percentile points) Statistically significant 


Mathematics achievement 20 classes/328 students +2 No 


Setting 

The study took place in five elementary schools in a metropolitan public school system in the 


southeastern United States. 

Study sample 

The study authors randomly assigned 20 female first-grade teachers to use either Peer- 
Assisted Learning Strategies or the standard math curriculum. Within each school, equal 
numbers of teachers were randomly assigned to each of the research conditions. The total 
sample included 10 teachers in each condition. The analysis sample size was 162 students in 
the intervention condition and 166 students in the comparison condition. 

Intervention 

group 

In this study, Peer-Assisted Learning Strategies supplemented the district’s core curriculum. 
Math Advantage. The study authors wanted to ensure that students in the intervention and 
comparison groups received the same amount of mathematics instruction during the 16 weeks 
when Peer-Assisted Learning Strategies was used with the intervention group. Therefore, 
the study’s implementation of Peer-Assisted Learning Strategies replaced some mathemat- 
ics instruction activities that intervention teachers would otherwise have used. Specifically, 
Peer-Assisted Learning Strategies was implemented in intervention classrooms for 30-minute 
sessions, three times a week, for 1 6 weeks. The study authors asked teachers to classify their 
students’ mathematics achievement at the beginning of the study. These classifications were 
used to pair low achieving students with high achieving students for the tutoring sessions. 
During the first part of each session, the stronger performing student was the coach (tutor); 
midway through each session, the stronger performing student switched to the tutee role. 
Every three weeks, teachers reassigned student pairs to increase exposure to different stu- 
dents and, after every third three-week cycle, teachers paired high achievers together. 

Comparison 

group 

Teachers assigned to the comparison condition used only Math Advantage Grade 1 in their 
classrooms. The curriculum includes teacher-directed lessons, student work with manipula- 
tives, and worksheets. According to the study authors, teachers in the comparison classrooms 
used the Math Advantage program for at least 90% of their mathematics instruction. Peer- 
tutoring activities (like those that are a central component of Peer-Assisted Learning Strate- 
gies) were rarely used in the comparison classrooms. 
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Outcomes and 
measurement 


Support for 
implementation 


student mathematics achievement was assessed using the Primary 1 and 2 ieveis of the SAT. 
The test administered inciuded 94 of the test’s 106 totai items. These 94 items were seiected to 
refiect the Peer-Assisted Learning Strategies curricuium and the district’s iarger core curricuium. 
The study authors divided the items into those aiigned with the Peer-Assisted Learning Strate- 
gies curricuium (72 items) and those unaiigned with the Peer-Assisted Learning Strategies cur- 
ricuium (22 items). Resuits were presented separateiy for the aiigned and unaiigned portions of 
the test. For a more detaiied description of these outcome measures, see Appendix B. 

Teachers attended a two-hour after-schooi workshop where they iearned about Peer-Assisted 
Learning Strategies and practiced impiementing the program. Teachers used Peer-Assisted 
Learning Strategies in their ciassrooms within one week after the workshop. Research assis- 
tants observed aii intervention sessions and deiivered Peer-Assisted Learning Strategies 
program materiais for the upcoming weeks. 
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Appendix B: Outcome measures for each domain 

Mathematics achievement 

Stanford Achievement Test 
(SAT): Aligned Items 

The SAT (Gardner, Rudman, Karlsen, & Merwin, 1987) measures mathematical knowledge and skills for grades K-3. The 
Primary 1 level measures knowledge and skills for K.5-1.9, while the Primary 2 level measures knowledge and skills for 
1.5-2. 9. There are 106 Items across Primary 1 and 2. The study authors asked tour first-grade teachers to characterize 
each SAT item as aligned or unaligned with the Peer-Assisted Learning Strategies curriculum. They identified 72 items as 
aligned with the Peer-Assisted Learning Strategies curriculum. The reported test-retest reliability of the aligned items was 
0.93 (as cited in Fuchs et at, 2002). 

SAT: Unaligned Items 

The study authors asked four first-grade teachers to characterize each SAT item as aligned or unaligned with the Peer- 
Assisted Learning Strategies curriculum. They identified 22 items ot the SAT as unaligned with the Peer-Assisted Learning 
Strategies curriculum. The reported test-retest reliability of the unaligned items was 0.78 (as cited in Fuchs et al., 2002). 
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Appendix C: Findings inciuded in the rating for the mathematics achievement domain 


Mean 

(standard deviation) WWC calculations 


Outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Fuchs et al., 2002^ 

Stanford Achievement Test 
(SAT): Aligned Items 

Grade 1 

20 Classes/ 
328 students 

47.56 

(16.98) 

45.07 

(18.43) 

2.49 

0.14 

-f6 

<0.02 

SAT: Unaligned Items 

Grade 1 

20 classes/ 
328 students 

16.55 

(3.19) 

16.65 

(3.65) 

-0.10 

-0.03 

-1 

0.76 

Domain average for mathematics achievement (Fuchs et al., 2002) 



0.06 

+2 

Not 


statistically 

significant 


Domain average for mathematics achievement across all studies 0.06 +2 na 


Table Notes: For mean difference, effecf size, and improvement index vaiues reported in the tabie, a positive number favors the intervention group and a negative number favors 
the comparison group. The effect size is a standardized measure of the effect of an intervention on student outcomes, representing the average change expected for aii students 
who are given the intervention (measured in standard deviations of the outcome measure). The improvement index is an aiternate presentation of the effect size, refiecting the 
change in an average student’s percentiie rank that can be expected if the student is given the intervention. The WWC-computed average effect size is a simpie average rounded 
to two decimai pieces; the average improvement index is caiculated from the average effect size. The statisticai significance of the study’s domain average was determined by the 
WWC. na = not applicabie. 

The p-vaiues presented here were reported in the originai study. For Fuchs et ai. (2002), a correction for ciustering was needed and resuited in significance ieveis that differ from 
those in the originai study. The WWC finds that the resuits for the SAT: Aligned Items outcome are not statisticaiiy significant after corrections. The intervention group means were 
adjusted for baseiine differences using a difference-in-differences adjustment. 
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Appendix D: Description of suppiementai findings for the mathematics achievement domain 


Mean 

(standard deviation) WWC calculations 


Outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Fuchs et al., 2002^ 

Stanford Achievement Test 
(SAT): Aligned Items 

High-achieving 

20 classes/ 
93 students 

55.38 

(13.48) 

52.19 

(15.22) 

3.19 

0.22 

-f9 

nr 

SAT: Unaligned Items 

High-achieving 

20 classes/ 
93 students 

18.69 

(1.70) 

18.78 

(2.03) 

-0.09 

-0.05 

-2 

nr 

SAT: Aligned Items 

Average- 

achieving 

20 classes/ 
152 students 

48.78 

(14.84) 

46.14 

(17.14) 

2.64 

0.16 

+7 

nr 

SAT: Unaligned Items 

Average- 

achieving 

20 classes/ 
152 students 

17.24 

(2.74) 

17.43 

(2.78) 

-0.19 

-0.07 

-3 

nr 

SAT: Aligned Items 

Low-achieving 

20 classes/ 
83 students 

38.89 

(17.81) 

35.51 

(17.16) 

3.38 

0.19 

-f8 

nr 

SAT: Unaligned Items 

Low-achieving 

20 classes/ 
83 students 

13.03 

(3.50) 

12.98 

(3.75) 

0.05 

0.01 

-fI 

nr 


Table Notes: The supplemental findings presented in this tabie are additionai findings from the study in this report that do not factor into the determination of the intervention 
rating. For mean difference, effect size, and improvement index vaiues reported in the tabie, a positive number favors the intervention group and a negative number favors fhe 
comparison group. The effecf size is a standardized measure of the effect of an infervention on sfudent oufcomes, representing the average change expected for aii sfudents who 
are given fhe intervenfion (measured in standard deviations of the outcome measure). The improvement index is an aiternate presentation of the effect size, refiecting the change 
in an average student’s percentiie rank that can be expected if the student is given the intervention, nr = not reported. 

“ For Fuchs et ai. (2002), corrections for ciustering and muitipie comparisons were needed. The intervention group means were adjusted for baseiine differences using a difference-in- 
differences adjustment. The p-vaiues are not provided for the specific contrasts of interest to the WWC. 
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Endnotes 

■' The descriptive information for this program was obtained from a publicly-available source: the program’s website (http:// 
kc.vanderbilt.edu/pals, downloaded August 2011). The WWC requests distributors review the program description sections for accu- 
racy from their perspective. The program description was provided to the distributor in August 201 1 ; however, the WWC received no 
response. Further verification of the accuracy of the descriptive information for this program is beyond the scope of this review. The 
literature search reflects documents publicly available by December 201 1 . 

^ The studies in this report were reviewed using the Evidence Standards from the WWC Procedures and Standards Handbook (version 
2.1), along with those described in the Elementary School Mathematics review protocol (version 2.0). The evidence presented in this 
report is based on available research. Findings and conclusions may change as new research becomes available. 

® For criteria used in the determination of the rating of effectiveness and extent of evidence, see the WWC Rating Criteria on p. 1 3. These 
improvement index numbers show the average and range of student-level improvement indices for all findings across the studies. 

Grade, delivery method, and program type refer to the studies that meet WWC evidence standards without or with reservations. 

Recommended Citation 

U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse. (2013, January). 
Elementary School Mathematics Intervention report: Peer-Assisted Learning Strategies. Retrieved from 
http://www.whatworks.ed.gov. 
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WWC Rating Criteria 

Criteria used to determine the rating of a study 


Study rating 

Criteria 

Meets WWC evidence standards 
without reservations 

A study that provides strong evidence for an intervention’s effectiveness, such as a weii-implemented RCT. 

Meets WWC evidence standards 
with reservations 

A study that provides weaker evidence for an intervention's effectiveness, such as a QED or an RCT with high 
attrition that has established equivaience of the analytic samples. 

Criteria used to determine the rating of effectiveness for an intervention 

Rating of effectiveness 

Criteria 

Positive effects 

Two or more studies show statisticaiiy significant positive effects, at least one of which met WWC evidence 
standards for a strong design, AND 

No studies show statistically significant or substantively important negative effects. 

Potentially positive effects 

At least one study shows a statistically significant or substantively important positive effect, AND 

No studies show a statistically significant or substantively important negative effect AND fewer or the same number 

of sfudies show indeterminafe effects than show statistically significant or substantively important positive effects. 

Mixed effects 

At least one study shows a statistically significant or substantively important positive effect AND at least one study 
shows a statistically significant or substantively important negative effect, but no more such studies than the number 
showing a statistically significant or substantively important positive effect, OR 

At least one study shows a statistically significant or substantively important effect AND more studies show an 
indeterminate effect than show a statistically significant or substantively important effect. 

Potentially negative effects 

One study shows a statistically significant or substantively important negative effect and no studies show 
a statistically significant or substantively important positive effect, OR 

Two or more studies show statistically significant or substantively important negative effects, at least one study 
shows a statistically significant or substantively important positive effect, and more studies show statistically 
significant or substantively important negative effects than show statistically significant or substantively important 
positive effects. 

Negative effects 

Two or more studies show statistically significant negative effects, at least one of which met WWC evidence 
standards for a strong design, AND 

No studies show statistically significant or substantively important positive effects. 

No discernible effects 

None of the studies shows a statistically significant or substantively important effect, either positive or negative. 

Criteria used to determine the extent of evidence for an intervention 

Extent of evidence 

Criteria 

Medium to large 

The domain includes more than one study, AND 
The domain includes more than one school, AND 

The domain findings are based on a total sample size of at least 350 students, OR, assuming 25 students in a class, 
a total of at least 14 classrooms across studies. 

Small 

The domain includes only one study, OR 
The domain includes only one school, OR 

The domain findings are based on a total sample size of fewer than 350 students, AND, assuming 25 students 
in a class, a total of fewer than 14 classrooms across studies. 
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Glossary of Terms 

Attrition 

Clustering adjustment 
Confounding factor 

Design 
Domain 
Effect size 

Eligibility 

Equivalence 

Extent of evidence 

Improvement index 

Multiple comparison 
adjustment 

Quasi-experimental 
design (QED) 

Randomized controlled 
trial (RCT) 

Rating of effectiveness 

Single-case design 
Standard deviation 


Statistical significance 


Substantively important 


Attrition occurs when an outcome variable is not avaiiabie for aii participants initiaiiy assigned 
to the intervention and comparison groups. The WWC considers the total attrition rate and 
the difference in attrition rates across groups within a study. 

If intervention assignment is made at a cluster level and the analysis is conducted at the student 
level, the WWC will adjust the statistical significance to account for this mismatch, if necessary. 

A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 

The design of a study is the method by which intervention and comparison groups were assigned. 
A domain is a group of closely related outcomes. 

The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 

A study is eligible for review and inclusion in this report if it falls within the scope of the 
review protocol and uses either an experimental or matched comparison group design. 

A demonstration that the analysis sample groups are similar on observed characteristics 
defined in the review area protocol. 

An indication of how much evidence supports the findings. The criteria for the extent 
of evidence levels are given in the WWC Rating Criteria on p. 13. 

Along a percentile distribution of students, the improvement index represents the gain 
or loss of the average student due to the intervention. As the average student starts at 
the 50th percentile, the measure ranges from -50 to +50. 

When a study includes multiple outcomes or comparison groups, the WWC will adjust 
the statistical significance to account for the multiple comparisons, if necessary. 

A quasi-experimental design (QED) is a research design in which subjects are assigned 
to intervention and comparison groups through a process that is not random. 

A randomized controlled trial (RCT) is an experiment in which investigators randomly assign 
eligible participants into intervention and comparison groups. 

The WWC rates the effects of an intervention in each domain based on the quality of the 
research design and the magnitude, statistical significance, and consistency in findings. The 
criteria for the ratings of effectiveness are given in the WWC Rating Criteria on p. 1 3'. 

A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 

The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample tend to be spread out over a large range of values. 

Statistical significance is the probability that the difference between groups is a result of 
chance rather than a real difference between the groups. The WWC labels a finding statistically 
significant if the likelihood that the difference is due to chance is less than 5% (p < 0.05). 

A substantively important finding is one that has an effect size of 0.25 or greater, regardless 
of statistical significance. 


Please see the WWC Procedures and Standards Handbook (version 2.1) for additional details. 
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